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CamegieMeUon 

Software  Engineering  Institute 

Background 

Case  studies  have  shown  that  properly  implemented 
systems  engineering  can  result  in  commensurate 
benefits 

Broadly  applicable  quantification  of  these  costs  and 
benefits  remains  elusive 

•  Complicated  by  the  lack  of  a  broadly  accepted  definition 
of  Systems  Engineering 

•  Insufficient  identification  and  tracking  of  Systems 
Engineering  costs  and  efforts 

•  Exacerbated  by  increasing  complexity  and  size  of 
systems  and  Systems  of  Systems 
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CamegieMeUon 

Software  Engineering  Institute 

The  Task 

The  Office  of  the  Under  Secretary  of  Defense  (AT&L)  has 
tasked  the  NDIA  Systems  Engineering  Division  to 
research  and  report  on  the  costs  and  benefits  associated 
with  Systems  Engineering  practices  in  the  acquisition 
and  /  or  development  of  military  systems. 


The  Systems  Engineering  Effectiveness  Committee 
(SEEC)  is  addressing  this  task  via  a  survey  of  program 
and  project  managers  across  the  defense  industry. 
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CamegieMeUon 

Software  Engineering  Institute 

Survey  Objective 

Identify  the  degree  of  correlation  between  the  use  of 
specific  systems  engineering  practices  and  activities  on 
projects,  and  quantitative  measures  of  project  /  program 
performance. 

Survey  Method 

Use  the  resources  of  NDIA  SE  Division  to  reach  a  broad 
constituency 

The  initial  survey  will  focus  on  industry  members  of  NDIA 
that  are  prime  contractors  and  subcontractors 

Collect  feedback  from  project  /  program  managers 
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CamegieMeUon 

Software  Engineering  Institute 

Survey  Development  Plan 

•  Define  the  goal 

•  Choose  the  population 

•  Define  the  means  to  assess  usage  of  SE  practices 

•  Define  the  measured  benefits  to  be  studied 

•  Develop  the  survey  instrument 

•  Execute  the  survey 

•  Analyze  the  results 

•  Report 

•  Plan  future  studies 
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CamegieMeUon 

Software  Engineering  Institute 

Step  1 : 

Define  the  Goal 

Identify  correlations  between  SE  practices  and  program 
performance 


Step  2: 

Choose  the  population 

Chosen  population  consists  of  contractors  and 
subcontractors  providing  products  to  the  DoD 
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Gxraegie  Mellon 

Software  Engineering  Institute 


Step  3: 

Define  assessment  of  SE  practices 


CMMI-SW/SE  vl.1 

•  22  Process  Areas 

•  157  Goals 

•  539  Practices 

•  402  Work  Products 


Systems 

Engineering 

Filter 


Size  Constraint 
Filter 


•  1 3  Process  Areas 

•  27  Goals 

•  75  Practices 
•185  Work  Products 


I 


1 0  Process  Areas 
19  Goals 
34  Practices 
63  Work  Products 
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CamegieMeUon 

Software  Engineering  Institute 

Step  4: 

Define  performance  measures 

Utilize  measures  common  to  many  organizations 

•  Earned  Value 

•  Award  Fees 

•  Technical  Requirements  Satisfaction 

•  Milestone  Satisfaction 

•  Problem  Reports 
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Gxraegie  Mellon 

Software  Engineering  Institute 


Step  5: 

Develop  the  survey  instrument 


Self-administration 

•  formatted  for  web-based 
deployment 

Confidentiality 

•  No  elicitation  of  identifying  data 

•  Anonymous  response  collection 

•  Responses  accessible  only  to 
authorized  SEI  staff 

Integrity 

•  Data  used  only  for  stated 
purpose 

•  No  attempt  to  extract 
identification  data 

Self-checking 
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Section  1  -  Characterization 


Characterization  of  the 

project  /  program  under 
consideration 
•Project  /  program 

-Size  -Stability 

-  Lifecycle  phase 
-Subcontracting 
-Application  domain 
-Customer  /  User 
-etc. 

•Organization 

-Size 

-Organizational  capability 

-  Related  experience 
-etc. 


Section  1:  Characterization 


The  objective  of  this  section  is  to  gather  information  to  characterize  the  project  under 
consideration.  This  information  will  assist  the  survey  analysts  in  categorizing  the  project, 
and  the  executing  organization  to  better  understand  your  responses. 


1.1  Project  -  information  to  characterize  the  specific  project  under  discussion. 

Size,  stability,  lifecycle  phase,  subcontracting,  and  application  domain  are 
among  the  parameters  used  for  program  characterization. 


1.1.1 


What  phases  of  the  integrated  product  lifecycle 
comprise  this  project  (check  all  that  apply),  and 
what  phase  are  you  presently  executing  (check  1)? 


IncliidejUn  project 
(c(he£K  alljthat  apply 

irrent 
ase 

(check  1) 

ConceptJRefmement 
Tephtfdlogy 
'evelopment  and 
Demonstration 
Development 
Manufacturing 
Verification 
Training 
Deployment 
Operation 
Support 
Disposal 


1.1.2 

What  is  the  current  total  contract  value  (US$)  of 
your  project? 

$ 

1.1.3 

What  was  the  initial  contract  value  (US$)  of  your 
project? 

$ 

1.1.4 

How  many  contract  change  orders  have  been 
received? 
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GamcgieMellan 

Software  Engineering  Institute 

Section  2:  SE  Evidence 


Process  definition 
Project  /program  planning 
Risk  management 
Requirements  development 
Requirements  management 
Trade  studies 
Interfaces 
Product  structure 
Product  integration 
Test  and  verification 
Project  /  program  reviews 
Validation 

Configuration  management 
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Gxraegie  Mellon 

Software  Engineering  Institute 


Section  3:  Performance  Metrics 


Earned  Value 
Award  fees 

Technical  requirements 
satisfaction 

Milestone  satisfaction 

Problem  reports 


Section  3:  Project  Performance  Metrics 


3.1  Earned  Value  Management  System  (EVMS) 


Rate  your  agreement  with  the 
following  statements 


m  a 'timely  manner  (i.e. 
^rffwithin  2  weeks)? 


3.1.3  The  requirement  to  track  and  report 
EVMS  data  is  levied  upon  the 
project’s  suppliers. 


3.1.4  Variance  thresholds  for  CPI  and  SPI 
variance  are  defined,  documented, 
and  used  to  determine  when 
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Gxraegie  Mellon 

Software  Engineering  Institute 


Step  6: 
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CamegieMeJkm 

Software  Engineering  Institute 

Step  7 : 

Analyze  the  results 

Partition  responses  based  on  project  characterizations 

Analyze  survey  responses  to  look  for  correlations  between 
the  SE  practices  and  the  chosen  metrics. 

Step  8: 

Report 

Summarize  survey  results  and  analysis  in  a  report. 

Step  9: 

Plan  future  studies 

Based  upon  the  findings  from  the  survey,  the  need  for 
additional  studies  may  be  defined. 
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GamcgieMellan 

Software  Engineering  Institute 

Status 

Survey  instrument  development  complete 
Web  deployment  complete 
Respondent  identification  in  progress 
Response  collection  through  Nov. 
Analysis  through  Dec.  and  Jan. 

Report  in  Feb. 
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GamcgieMellan 

Software  Engineering  Institute 

SE  Effectiveness  Committee 


Dennis  Aheam 
David  P.  Ball 
Thomas  Christian 
Greg  DiBennedetto 
Terry  Doran 
Donald  J.  Gantzer 
Ellis  Hitte 
Ed  Kunay 
Gordon  F.  Neary* 
Brooks  Nolan 
Rusty  Rentsch 
Rex  Sallade 
Jack  Stockdale 
Ruth  Wuenschel 


Marvin  Anthony 
A1  Brown* 

Jack  Crowley 
Jim  Dietz 
Joseph  Elm 
Dennis  Goldenson 
James  Holton 
Jeff  Loren 
Brad  Nelson* 
Michael  Persson* 
Paul  Robitaille 
Jay  R.  Schrand 
Jason  Stripinis 
Brenda  Zettervall 


Ben  Badami 
A1  Bruns 
John  Colombi 
Brian  Donahue 
John  P.  Gaddie 
Dennis  E.  Hecht 
George  Kailiwai 
John  Miller 
Rick  Neupert 
Bob  Rassa 
Garry  Roedler 
Sarah  Sheard 
Mike  Ucchino* 
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Conclusion 


Contact  information 

•  Joseph  P.  Elm  ielm@sei.cmu.edu 
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=-  CarnegieMdlon 

Software  Engineering  Institute 

Target  Audience 


AAI  Corp. 

•  Gestalt,  LLC 

•  Scientific  Solutions,  Inc. 

Alion  Science  &  Technology 

•  Harris  Corp. 

•  SI  International 

Allied-Signal 

•  Honeywell 

•  Simulation  Strategies  Inc. 

Anteon  Corp 

•  Hughes  Space  & 
Communications 

•  Southwest  Research  Institute 

AT&T 

•  Impact  Technologies  LLC 

•  SRA  International 

BAE  Systems 

•  ITT  Industries 

•  Support  Systems  Associates  Inc. 

BBN  Technologies 

•  Jacobs  Sverdrup 

•  Systems  &  Electronics,  Inc. 

Boeing 

•  L-3  Communications 

•  TERADYNE,  Inc. 

Computer  Sciences  Corp. 

•  Lockheed  Martin 

•  Titan  Systems  Co.  (AverStar  Group) 

Concurrent  Technologies  Corp. 

•  Motorola 

•  Trident  Systems,  Inc. 

DCS  Corp. 

•  Northrop  Grumman 

•  TRW  Inc. 

DRS  Technologies 

•  Orbital  Sciences  Corp. 

•  United  Defense  LP 

Foster-Miller  Inc. 

•  Raytheon 

•  United  Technologies 

GE 

•  Rockwell  Collins 

•  Virtual  Technology  Corp. 

General  Dynamics 

•  SAIC 

•  Vitech  Corp. 

Selection  criteria: ! 


Active  in  NDIA  SED 

Contractors  delivering  products  to  the  government 


Need  Point-of-Contact  (Focal)  from  each  company  to  expedite 

survey  deployment. 
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